Bayes Factor

Bayes Factor

From Keysers et al. (2020)

Bayes Factor and multiple testing

\text{Pr(at least one } BF \geq 3)

N.B. Results change depending on priors and threshold.

Posterior odds and multiple testing

Researchers often interpret BF as posterior odds (Hoijtink et al., 2016; Tendeiro et al., 2024; Tendeiro & Kiers, 2019).

\text{Posterior odds, } \frac{Pr(\mathcal{H_1} | data)}{Pr(\mathcal{H_0} | data)} = \frac{Pr(\mathcal{H_1})}{Pr(\mathcal{H_0})} \times \frac{p(data | \mathcal{H_1})}{p(data | \mathcal{H_0})}

Multiplicity adjustment can be incorporated by modifying the prior odds \frac{Pr(\mathcal{H}_1)}{Pr(\mathcal{H}_0)} based on the number of hypotheses tested, which produces posterior odds that control for multiple testing.

Bayesian estimation

The prior (grey) represents our initial guess about plausible values; after observing data, it updates to the posterior (blue), which narrows around the most likely value.

Bayesian estimation concrete example

Suppose we want to estimate the Stroop effect (0.060–0.120s). We can adopt priors centered at zero with varying standard deviations (τ = 0.1, 0.25, 0.5) to reflect increasing skepticism.

Probability of Direction

p_d = \max(Pr({\hat{\beta}} < 0), Pr({\hat{\beta}} > 0))

Probability of Direction and Multiplicity

fwer = \text{Pr(at least one } P_d \geq 0.975)

The joint posterior approach

You want to test whether an intervention improves verbal and/or visuospatial abilities, identify which domains improve. Improvement in at least one domain is more probable than improvement in both.

Example of hypothesis testing

Questions for You

Who here has used Bayesian methods? What was your experience?

What are the main barriers to correct statistical understanding and practice in your field? What would help us improve?

We often test multiple hypotheses in a single study. What drives this practice in your research? Do the benefits outweigh the costs?

H_0 = 0, H_1 = Cauchy(scale = \sqrt{2}/2)

Between-models priors (Jeffreys, 1938)

• m alternative hypotheses.

• Let \mathcal{H}_A be the event that at least one alternative hypothesis is true.

• We assign equal prior probability to the event \mathcal{H}_A and to its complement \mathcal{H}_0 (the global null, all false):

Pr(\mathcal{H}_0) = Pr(\mathcal{H}_A) = \frac{1}{2}

This ensures that the probability of all nulls being true simultaneously and the probability of at least one alternative being true both equal 0.5.

Westfall’s approach for dependent tests

For multiple comparisons among m means, Westfall et al. (1997) developed a solution for the \binom{m}{2} possible pairwise comparisons involving \mu_1, \mu_2, \ldots, \mu_m, where \theta_{ij} represents the difference \mu_i - \mu_j. Their hierarchical prior structure specifies:

\mu_i \begin{cases} \equiv \mu & \text{with probability } \lambda, \\ \sim G & \text{with probability } 1-\lambda, \end{cases}

with G(\cdot) being a continuous distribution.

Under this specification, the probability that any two specific means equal \mu is \lambda^2, and \text{Pr}(\text{all }\theta_{ij} = 0) = \lambda^m. This dependency structure yields a modified adjustment k = 1-0.5^{2/m}, replacing the independence-based correction (resembles previously described mixture models).

Note: Implemented in JASP for post-hoc testing (for details, see de Jong, 2019).

Bayesian estimation prior precision